Time Series Forecasting Model Tutorial

This notebook guides model developers through the process of automatically documenting and testing time series forecasting models. It shows you how to use the ValidMind Developer Framework to import and prepare data and before running a data validation test suite, followed by loading a pre-trained model and running a model validation test suite.

As part of the notebook, you will learn how to:

  • Step 1: Import raw data
  • Step 3: Run data validation test suite on raw data
  • Step 4: Preprocess data
  • Step 5: Run data validation test suite on processed data
  • Step 6: Load pre-trained models
  • Step 7: Run model validation test suite on models

ValidMind at a glance

ValidMind’s platform enables organizations to identify, document, and manage model risks for all types of models, including AI/ML models, LLMs, and statistical models. As a model developer, you use the ValidMind Developer Framework to automate documentation and validation tests, and then use the ValidMind AI Risk Platform UI to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators.

If this is your first time trying out ValidMind, we recommend going through the following resources first:

Before you begin

New to ValidMind?

To access the ValidMind Platform UI, you’ll need an account.

Signing up is FREE — Create your account.

If you encounter errors due to missing modules in your Python environment, install the modules with pip install, and then re-run the notebook. For more help, refer to Installing Python Modules.

Install the client library

The client library provides Python support for the ValidMind Developer Framework. To install it:

%pip install -q validmind
WARNING: You are using pip version 22.0.3; however, version 24.0 is available.
You should consider upgrading via the '/Users/andres/code/validmind-sdk/.venv/bin/python3 -m pip install --upgrade pip' command.
Note: you may need to restart the kernel to use updated packages.

Initialize the client library

ValidMind generates a unique code snippet for each registered model to connect with your developer environment. You initialize the client library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.

Get your code snippet:

  1. In a browser, log into the Platform UI.

  2. In the left sidebar, navigate to Model Inventory and click + Register new model.

  3. Enter the model details, making sure to select Time Series Forecasting as the template and Credit Risk - Underwriting - Loan as the use case, and click Continue. (Need more help?)

  4. Go to Getting Started and click Copy snippet to clipboard.

Next, replace this placeholder with your own code snippet:

# Replace with your code snippet

import validmind as vm

vm.init(
    api_host="...",
    api_key="...",
    api_secret="...",
    project="...",
)
2024-04-10 17:28:10,149 - INFO(validmind.api_client): Connected to ValidMind. Project: [Int. Tests] Time Series - Initial Validation (cltnl8c7v001j1omgyzmjrzhj)

Explore available test suites

In this notebook we will run a collection of test suites that are available in the ValidMind Developer Framework. Test suites group together a collection of tests that are relevant for a specific use case. In our case, we will run test different test suites for time series forecasting models. Once a test suite runs successfully, its results will be automatically uploaded to the ValidMind platform.

vm.test_suites.list_suites()
ID Name Description Tests
classifier_model_diagnosis ClassifierDiagnosis Test suite for sklearn classifier model diagnosis tests validmind.model_validation.sklearn.OverfitDiagnosis, validmind.model_validation.sklearn.WeakspotsDiagnosis, validmind.model_validation.sklearn.RobustnessDiagnosis
classifier_full_suite ClassifierFullSuite Full test suite for binary classification models. validmind.data_validation.DatasetDescription, validmind.data_validation.DescriptiveStatistics, validmind.data_validation.PearsonCorrelationMatrix, validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.HighCardinality, validmind.data_validation.HighPearsonCorrelation, validmind.data_validation.MissingValues, validmind.data_validation.Skewness, validmind.data_validation.UniqueRows, validmind.data_validation.TooManyZeroValues, validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.ConfusionMatrix, validmind.model_validation.sklearn.ClassifierPerformance, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.PrecisionRecallCurve, validmind.model_validation.sklearn.ROCCurve, validmind.model_validation.sklearn.PopulationStabilityIndex, validmind.model_validation.sklearn.SHAPGlobalImportance, validmind.model_validation.sklearn.MinimumAccuracy, validmind.model_validation.sklearn.MinimumF1Score, validmind.model_validation.sklearn.MinimumROCAUCScore, validmind.model_validation.sklearn.TrainingTestDegradation, validmind.model_validation.sklearn.ModelsPerformanceComparison, validmind.model_validation.sklearn.OverfitDiagnosis, validmind.model_validation.sklearn.WeakspotsDiagnosis, validmind.model_validation.sklearn.RobustnessDiagnosis
classifier_metrics ClassifierMetrics Test suite for sklearn classifier metrics validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.ConfusionMatrix, validmind.model_validation.sklearn.ClassifierPerformance, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.PrecisionRecallCurve, validmind.model_validation.sklearn.ROCCurve, validmind.model_validation.sklearn.PopulationStabilityIndex, validmind.model_validation.sklearn.SHAPGlobalImportance
classifier_model_validation ClassifierModelValidation Test suite for binary classification models. validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.ConfusionMatrix, validmind.model_validation.sklearn.ClassifierPerformance, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.PrecisionRecallCurve, validmind.model_validation.sklearn.ROCCurve, validmind.model_validation.sklearn.PopulationStabilityIndex, validmind.model_validation.sklearn.SHAPGlobalImportance, validmind.model_validation.sklearn.MinimumAccuracy, validmind.model_validation.sklearn.MinimumF1Score, validmind.model_validation.sklearn.MinimumROCAUCScore, validmind.model_validation.sklearn.TrainingTestDegradation, validmind.model_validation.sklearn.ModelsPerformanceComparison, validmind.model_validation.sklearn.OverfitDiagnosis, validmind.model_validation.sklearn.WeakspotsDiagnosis, validmind.model_validation.sklearn.RobustnessDiagnosis
classifier_validation ClassifierPerformance Test suite for sklearn classifier models validmind.model_validation.sklearn.MinimumAccuracy, validmind.model_validation.sklearn.MinimumF1Score, validmind.model_validation.sklearn.MinimumROCAUCScore, validmind.model_validation.sklearn.TrainingTestDegradation, validmind.model_validation.sklearn.ModelsPerformanceComparison
cluster_full_suite ClusterFullSuite Full test suite for clustering models. validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.HomogeneityScore, validmind.model_validation.sklearn.CompletenessScore, validmind.model_validation.sklearn.VMeasure, validmind.model_validation.sklearn.AdjustedRandIndex, validmind.model_validation.sklearn.AdjustedMutualInformation, validmind.model_validation.sklearn.FowlkesMallowsScore, validmind.model_validation.sklearn.ClusterPerformanceMetrics, validmind.model_validation.sklearn.ClusterCosineSimilarity, validmind.model_validation.sklearn.SilhouettePlot, validmind.model_validation.ClusterSizeDistribution, validmind.model_validation.sklearn.HyperParametersTuning, validmind.model_validation.sklearn.KMeansClustersOptimization
cluster_metrics ClusterMetrics Test suite for sklearn clustering metrics validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.HomogeneityScore, validmind.model_validation.sklearn.CompletenessScore, validmind.model_validation.sklearn.VMeasure, validmind.model_validation.sklearn.AdjustedRandIndex, validmind.model_validation.sklearn.AdjustedMutualInformation, validmind.model_validation.sklearn.FowlkesMallowsScore, validmind.model_validation.sklearn.ClusterPerformanceMetrics, validmind.model_validation.sklearn.ClusterCosineSimilarity, validmind.model_validation.sklearn.SilhouettePlot
cluster_performance ClusterPerformance Test suite for sklearn cluster performance validmind.model_validation.ClusterSizeDistribution
embeddings_full_suite EmbeddingsFullSuite Full test suite for embeddings models. validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.embeddings.DescriptiveAnalytics, validmind.model_validation.embeddings.CosineSimilarityDistribution, validmind.model_validation.embeddings.ClusterDistribution, validmind.model_validation.embeddings.EmbeddingsVisualization2D, validmind.model_validation.embeddings.StabilityAnalysisRandomNoise, validmind.model_validation.embeddings.StabilityAnalysisSynonyms, validmind.model_validation.embeddings.StabilityAnalysisKeyword, validmind.model_validation.embeddings.StabilityAnalysisTranslation
embeddings_metrics EmbeddingsMetrics Test suite for embeddings metrics validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.embeddings.DescriptiveAnalytics, validmind.model_validation.embeddings.CosineSimilarityDistribution, validmind.model_validation.embeddings.ClusterDistribution, validmind.model_validation.embeddings.EmbeddingsVisualization2D
embeddings_model_performance EmbeddingsPerformance Test suite for embeddings model performance validmind.model_validation.embeddings.StabilityAnalysisRandomNoise, validmind.model_validation.embeddings.StabilityAnalysisSynonyms, validmind.model_validation.embeddings.StabilityAnalysisKeyword, validmind.model_validation.embeddings.StabilityAnalysisTranslation
hyper_parameters_optimization KmeansParametersOptimization Test suite for sklearn hyperparameters optimization validmind.model_validation.sklearn.HyperParametersTuning, validmind.model_validation.sklearn.KMeansClustersOptimization
llm_classifier_full_suite LLMClassifierFullSuite Full test suite for LLM classification models. validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.nlp.StopWords, validmind.data_validation.nlp.Punctuations, validmind.data_validation.nlp.CommonWords, validmind.data_validation.nlp.TextDescription, validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.ConfusionMatrix, validmind.model_validation.sklearn.ClassifierPerformance, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.PrecisionRecallCurve, validmind.model_validation.sklearn.ROCCurve, validmind.model_validation.sklearn.PopulationStabilityIndex, validmind.model_validation.sklearn.SHAPGlobalImportance, validmind.model_validation.sklearn.MinimumAccuracy, validmind.model_validation.sklearn.MinimumF1Score, validmind.model_validation.sklearn.MinimumROCAUCScore, validmind.model_validation.sklearn.TrainingTestDegradation, validmind.model_validation.sklearn.ModelsPerformanceComparison, validmind.model_validation.sklearn.OverfitDiagnosis, validmind.model_validation.sklearn.WeakspotsDiagnosis, validmind.model_validation.sklearn.RobustnessDiagnosis, validmind.prompt_validation.Bias, validmind.prompt_validation.Clarity, validmind.prompt_validation.Conciseness, validmind.prompt_validation.Delimitation, validmind.prompt_validation.NegativeInstruction, validmind.prompt_validation.Robustness, validmind.prompt_validation.Specificity
prompt_validation PromptValidation Test suite for prompt validation validmind.prompt_validation.Bias, validmind.prompt_validation.Clarity, validmind.prompt_validation.Conciseness, validmind.prompt_validation.Delimitation, validmind.prompt_validation.NegativeInstruction, validmind.prompt_validation.Robustness, validmind.prompt_validation.Specificity
nlp_classifier_full_suite NLPClassifierFullSuite Full test suite for NLP classification models. validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.nlp.StopWords, validmind.data_validation.nlp.Punctuations, validmind.data_validation.nlp.CommonWords, validmind.data_validation.nlp.TextDescription, validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.ConfusionMatrix, validmind.model_validation.sklearn.ClassifierPerformance, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.PrecisionRecallCurve, validmind.model_validation.sklearn.ROCCurve, validmind.model_validation.sklearn.PopulationStabilityIndex, validmind.model_validation.sklearn.SHAPGlobalImportance, validmind.model_validation.sklearn.MinimumAccuracy, validmind.model_validation.sklearn.MinimumF1Score, validmind.model_validation.sklearn.MinimumROCAUCScore, validmind.model_validation.sklearn.TrainingTestDegradation, validmind.model_validation.sklearn.ModelsPerformanceComparison, validmind.model_validation.sklearn.OverfitDiagnosis, validmind.model_validation.sklearn.WeakspotsDiagnosis, validmind.model_validation.sklearn.RobustnessDiagnosis
regression_metrics RegressionMetrics Test suite for performance metrics of regression metrics validmind.data_validation.DatasetSplit, validmind.model_validation.ModelMetadata, validmind.model_validation.sklearn.PermutationFeatureImportance
regression_model_description RegressionModelDescription Test suite for performance metric of regression model of statsmodels library validmind.data_validation.DatasetSplit, validmind.model_validation.ModelMetadata
regression_models_evaluation RegressionModelsEvaluation Test suite for metrics comparison of regression model of statsmodels library validmind.model_validation.statsmodels.RegressionModelsCoeffs, validmind.model_validation.statsmodels.RegressionModelsPerformance
regression_models_comparison RegressionModelsComparison Test suite for regression models performance comparison validmind.model_validation.sklearn.RegressionModelsPerformanceComparison
regression_full_suite RegressionFullSuite Full test suite for regression models. validmind.data_validation.DatasetDescription, validmind.data_validation.DescriptiveStatistics, validmind.data_validation.PearsonCorrelationMatrix, validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.HighCardinality, validmind.data_validation.HighPearsonCorrelation, validmind.data_validation.MissingValues, validmind.data_validation.Skewness, validmind.data_validation.UniqueRows, validmind.data_validation.TooManyZeroValues, validmind.data_validation.DatasetSplit, validmind.model_validation.ModelMetadata, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.RegressionErrors, validmind.model_validation.sklearn.RegressionR2Square, validmind.model_validation.sklearn.RegressionModelsPerformanceComparison
regression_performance RegressionPerformance Test suite for regression model performance validmind.model_validation.sklearn.RegressionErrors, validmind.model_validation.sklearn.RegressionR2Square
summarization_metrics SummarizationMetrics Test suite for Summarization metrics validmind.model_validation.RougeMetrics, validmind.model_validation.TokenDisparity, validmind.model_validation.BleuScore, validmind.model_validation.BertScore, validmind.model_validation.ContextualRecall
tabular_dataset TabularDataset Test suite for tabular datasets. validmind.data_validation.DatasetDescription, validmind.data_validation.DescriptiveStatistics, validmind.data_validation.PearsonCorrelationMatrix, validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.HighCardinality, validmind.data_validation.HighPearsonCorrelation, validmind.data_validation.MissingValues, validmind.data_validation.Skewness, validmind.data_validation.UniqueRows, validmind.data_validation.TooManyZeroValues
tabular_dataset_description TabularDatasetDescription Test suite to extract metadata and descriptive statistics from a tabular dataset validmind.data_validation.DatasetDescription, validmind.data_validation.DescriptiveStatistics, validmind.data_validation.PearsonCorrelationMatrix
tabular_data_quality TabularDataQuality Test suite for data quality on tabular datasets validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.HighCardinality, validmind.data_validation.HighPearsonCorrelation, validmind.data_validation.MissingValues, validmind.data_validation.Skewness, validmind.data_validation.UniqueRows, validmind.data_validation.TooManyZeroValues
text_data_quality TextDataQuality Test suite for data quality on text data validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.nlp.StopWords, validmind.data_validation.nlp.Punctuations, validmind.data_validation.nlp.CommonWords, validmind.data_validation.nlp.TextDescription
time_series_data_quality TimeSeriesDataQuality Test suite for data quality on time series datasets validmind.data_validation.TimeSeriesOutliers, validmind.data_validation.TimeSeriesMissingValues, validmind.data_validation.TimeSeriesFrequency
time_series_dataset TimeSeriesDataset Test suite for time series datasets. validmind.data_validation.TimeSeriesOutliers, validmind.data_validation.TimeSeriesMissingValues, validmind.data_validation.TimeSeriesFrequency, validmind.data_validation.TimeSeriesLinePlot, validmind.data_validation.TimeSeriesHistogram, validmind.data_validation.ACFandPACFPlot, validmind.data_validation.SeasonalDecompose, validmind.data_validation.AutoSeasonality, validmind.data_validation.AutoStationarity, validmind.data_validation.RollingStatsPlot, validmind.data_validation.AutoAR, validmind.data_validation.AutoMA, validmind.data_validation.ScatterPlot, validmind.data_validation.LaggedCorrelationHeatmap, validmind.data_validation.EngleGrangerCoint, validmind.data_validation.SpreadPlot
time_series_model_validation TimeSeriesModelValidation Test suite for time series model validation. validmind.data_validation.DatasetSplit, validmind.model_validation.ModelMetadata, validmind.model_validation.statsmodels.RegressionModelsCoeffs, validmind.model_validation.statsmodels.RegressionModelsPerformance, validmind.model_validation.statsmodels.RegressionModelForecastPlotLevels, validmind.model_validation.statsmodels.RegressionModelSensitivityPlot
time_series_multivariate TimeSeriesMultivariate This test suite provides a preliminary understanding of the features and relationship in multivariate dataset. It presents various multivariate visualizations that can help identify patterns, trends, and relationships between pairs of variables. The visualizations are designed to explore the relationships between multiple features simultaneously. They allow you to quickly identify any patterns or trends in the data, as well as any potential outliers or anomalies. The individual feature distribution can also be explored to provide insight into the range and frequency of values observed in the data. This multivariate analysis test suite aims to provide an overview of the data structure and guide further exploration and modeling. validmind.data_validation.ScatterPlot, validmind.data_validation.LaggedCorrelationHeatmap, validmind.data_validation.EngleGrangerCoint, validmind.data_validation.SpreadPlot
time_series_sensitivity TimeSeriesSensitivity This test suite performs sensitivity analysis on a statsmodels OLS linear regression model by applying distinct shocks to each input variable individually and then computing the model's predictions. The aim of this test suite is to investigate the model's responsiveness to variations in its inputs. By juxtaposing the model's predictions under baseline and shocked conditions, users can visually evaluate the sensitivity of the model to changes in each variable. This kind of analysis can also shed light on potential model limitations, including over-reliance on specific variables or insufficient responsiveness to changes in inputs. As a result, this test suite can provide insights that may be beneficial for refining the model structure, improving its robustness, and ensuring a more reliable prediction performance. validmind.model_validation.statsmodels.RegressionModelSensitivityPlot
time_series_univariate TimeSeriesUnivariate This test suite provides a preliminary understanding of the target variable(s) used in the time series dataset. It visualizations that present the raw time series data and a histogram of the target variable(s). The raw time series data provides a visual inspection of the target variable's behavior over time. This helps to identify any patterns or trends in the data, as well as any potential outliers or anomalies. The histogram of the target variable displays the distribution of values, providing insight into the range and frequency of values observed in the data. validmind.data_validation.TimeSeriesLinePlot, validmind.data_validation.TimeSeriesHistogram, validmind.data_validation.ACFandPACFPlot, validmind.data_validation.SeasonalDecompose, validmind.data_validation.AutoSeasonality, validmind.data_validation.AutoStationarity, validmind.data_validation.RollingStatsPlot, validmind.data_validation.AutoAR, validmind.data_validation.AutoMA

For our example use case we will run the following test suites:

  • time_series_dataset
  • time_series_model_validation

Step 1: Import raw data

Import FRED dataset

Federal Reserve Economic Data, or FRED, is a comprehensive database maintained by the Federal Reserve Bank of St. Louis. It offers a wide array of economic data from various sources, including U.S. government agencies and international organizations. The dataset encompasses numerous economic indicators across various categories such as employment, consumer price indices, money supply, and gross domestic product, among others.

FRED provides a valuable resource for researchers, policymakers, and anyone interested in understanding economic trends and conducting economic analysis. The platform also includes tools for data visualization, which can help users interpret complex economic data and identify trends over time.

The following code snippet imports a sample FRED dataset into a Pandas dataframe:

from validmind.datasets.regression import fred as demo_dataset

target_column = demo_dataset.target_column
feature_columns = demo_dataset.feature_columns

df = demo_dataset.load_data()
df.tail(10)
MORTGAGE30US FEDFUNDS GS10 UNRATE
DATE
2023-03-02 6.65 NaN NaN NaN
2023-03-09 6.73 NaN NaN NaN
2023-03-16 6.60 NaN NaN NaN
2023-03-23 6.42 NaN NaN NaN
2023-03-30 6.32 NaN NaN NaN
2023-04-01 NaN NaN 3.46 NaN
2023-04-06 6.28 NaN NaN NaN
2023-04-13 6.27 NaN NaN NaN
2023-04-20 6.39 NaN NaN NaN
2023-04-27 6.43 NaN NaN NaN

Step 2: Run data validation test suite on raw data

Explore the time series dataset test suites

Let’s see what tests are included on each test suite:

vm.test_suites.describe_suite("time_series_data_quality")
ID Name Description Tests
time_series_data_quality TimeSeriesDataQuality Test suite for data quality on time series datasets validmind.data_validation.TimeSeriesOutliers, validmind.data_validation.TimeSeriesMissingValues, validmind.data_validation.TimeSeriesFrequency
vm.test_suites.describe_suite("time_series_univariate")
ID Name Description Tests
time_series_univariate TimeSeriesUnivariate This test suite provides a preliminary understanding of the target variable(s) used in the time series dataset. It visualizations that present the raw time series data and a histogram of the target variable(s). The raw time series data provides a visual inspection of the target variable's behavior over time. This helps to identify any patterns or trends in the data, as well as any potential outliers or anomalies. The histogram of the target variable displays the distribution of values, providing insight into the range and frequency of values observed in the data. validmind.data_validation.TimeSeriesLinePlot, validmind.data_validation.TimeSeriesHistogram, validmind.data_validation.ACFandPACFPlot, validmind.data_validation.SeasonalDecompose, validmind.data_validation.AutoSeasonality, validmind.data_validation.AutoStationarity, validmind.data_validation.RollingStatsPlot, validmind.data_validation.AutoAR, validmind.data_validation.AutoMA

Initialize the dataset

Use the ValidMind Developer Framework to initialize the dataset object:

vm_dataset = vm.init_dataset(
    input_id="raw_dataset",
    dataset=df,
    target_column=demo_dataset.target_column,
)
2024-04-10 17:28:10,243 - INFO(validmind.client): Pandas dataset detected. Initializing VM Dataset instance...

Run time series dataset test suite on raw dataset

Next, use the ValidMind Developer Framework to run the test suite for time series datasets:

config = {
    # TIME SERIES DATA QUALITY PARAMS
    "validmind.data_validation.TimeSeriesOutliers": {"params": {"zscore_threshold": 3}},
    "validmind.data_validation.TimeSeriesMissingValues": {
        "params": {"min_threshold": 2}
    },
    # TIME SERIES UNIVARIATE PARAMS
    "validmind.data_validation.RollingStatsPlot": {"params": {"window_size": 12}},
    "validmind.data_validation.SeasonalDecompose": {
        "params": {"seasonal_model": "additive"}
    },
    "validmind.data_validation.AutoSeasonality": {
        "params": {"min_period": 1, "max_period": 3}
    },
    "validmind.data_validation.AutoStationarity": {
        "params": {"max_order": 3, "threshold": 0.05}
    },
    "validmind.data_validation.AutoAR": {"params": {"max_ar_order": 2}},
    "validmind.data_validation.AutoMA": {"params": {"max_ma_order": 2}},
    # TIME SERIES MULTIVARIATE PARAMS
    "validmind.data_validation.LaggedCorrelationHeatmap": {
        "params": {
            "target_col": demo_dataset.target_column,
            "independent_vars": demo_dataset.feature_columns,
        }
    },
    "validmind.data_validation.EngleGrangerCoint": {"params": {"threshold": 0.05}},
}

full_suite = vm.run_test_suite(
    "time_series_dataset",
    inputs={
        "dataset": vm_dataset,
    },
    config=config,
)
No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/graphics/tsaplots.py:348: FutureWarning:

The default method 'yw' can produce PACF values outside of the [-1,1] interval. After 0.13, the default will change tounadjusted Yule-Walker ('ywm'). You can use this method now by setting method='ywm'.
/Users/andres/code/validmind-sdk/validmind/tests/data_validation/SeasonalDecompose.py:202: UserWarning:

No frequency could be inferred for variable 'MORTGAGE30US'. Skipping seasonal decomposition and plots for this variable.

2024-04-10 17:28:12,609 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of FEDFUNDS: MS
2024-04-10 17:28:12,609 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of FEDFUNDS: MS
2024-04-10 17:28:12,988 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of GS10: MS
2024-04-10 17:28:12,988 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of GS10: MS
2024-04-10 17:28:13,124 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of UNRATE: MS
2024-04-10 17:28:13,124 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of UNRATE: MS
No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
2024-04-10 17:28:13,634 - WARNING(validmind.tests.data_validation.AutoAR): Warning: MORTGAGE30US is not stationary. Results may be inaccurate.
2024-04-10 17:28:13,634 - WARNING(validmind.tests.data_validation.AutoAR): Warning: MORTGAGE30US is not stationary. Results may be inaccurate.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

2024-04-10 17:28:13,739 - WARNING(validmind.tests.data_validation.AutoAR): Warning: GS10 is not stationary. Results may be inaccurate.
2024-04-10 17:28:13,739 - WARNING(validmind.tests.data_validation.AutoAR): Warning: GS10 is not stationary. Results may be inaccurate.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.
2024-04-10 17:28:13,972 - WARNING(validmind.tests.data_validation.AutoMA): Warning: MORTGAGE30US is not stationary. Results may be inaccurate.
2024-04-10 17:28:13,972 - WARNING(validmind.tests.data_validation.AutoMA): Warning: MORTGAGE30US is not stationary. Results may be inaccurate.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:

Non-invertible starting MA parameters found. Using zeros as starting parameters.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:

Non-invertible starting MA parameters found. Using zeros as starting parameters.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:

Non-invertible starting MA parameters found. Using zeros as starting parameters.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:

Non-invertible starting MA parameters found. Using zeros as starting parameters.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:

Non-invertible starting MA parameters found. Using zeros as starting parameters.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:

Non-invertible starting MA parameters found. Using zeros as starting parameters.
2024-04-10 17:28:15,882 - WARNING(validmind.tests.data_validation.AutoMA): Warning: GS10 is not stationary. Results may be inaccurate.
2024-04-10 17:28:15,882 - WARNING(validmind.tests.data_validation.AutoMA): Warning: GS10 is not stationary. Results may be inaccurate.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:

Non-invertible starting MA parameters found. Using zeros as starting parameters.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:

Non-invertible starting MA parameters found. Using zeros as starting parameters.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:

Non-invertible starting MA parameters found. Using zeros as starting parameters.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:

Non-invertible starting MA parameters found. Using zeros as starting parameters.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/statespace/sarimax.py:978: UserWarning:

Non-invertible starting MA parameters found. Using zeros as starting parameters.

Step 3: Preprocess data

Handle frequencies, missing values and stationairty

# Sample frequencies to Monthly
resampled_df = df.resample("MS").last()

#  Remove all missing values
nona_df = resampled_df.dropna()

#  Take the first different across all variables
preprocessed_df = nona_df.diff().dropna()

Step 4: Run data validation test suite on processed data

vm_dataset = vm.init_dataset(
    input_id="preprocess_dataset",
    dataset=preprocessed_df,
    target_column=demo_dataset.target_column,
)

full_suite = vm.run_test_suite(
    "time_series_dataset",
    inputs={"dataset": vm_dataset},
    config=config,
)
2024-04-10 17:28:33,643 - INFO(validmind.client): Pandas dataset detected. Initializing VM Dataset instance...
No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/graphics/tsaplots.py:348: FutureWarning:

The default method 'yw' can produce PACF values outside of the [-1,1] interval. After 0.13, the default will change tounadjusted Yule-Walker ('ywm'). You can use this method now by setting method='ywm'.
2024-04-10 17:28:34,844 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of MORTGAGE30US: MS
2024-04-10 17:28:34,844 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of MORTGAGE30US: MS
2024-04-10 17:28:35,127 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of FEDFUNDS: MS
2024-04-10 17:28:35,127 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of FEDFUNDS: MS
2024-04-10 17:28:35,350 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of GS10: MS
2024-04-10 17:28:35,350 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of GS10: MS
2024-04-10 17:28:35,483 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of UNRATE: MS
2024-04-10 17:28:35,483 - INFO(validmind.tests.data_validation.SeasonalDecompose): Frequency of UNRATE: MS
No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.
/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

/Users/andres/code/validmind-sdk/.venv/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:471: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

Step 5: Load pre-trained models

Load pre-trained models

from validmind.datasets.regression import fred as demo_dataset

model_A, train_df_A, test_df_A = demo_dataset.load_model("fred_loan_rates_model_3")
model_B, train_df_B, test_df_B = demo_dataset.load_model("fred_loan_rates_model_4")

Initialize Validmind models

# Initialize training and testing datasets for model A
vm_train_ds_A = vm.init_dataset(
    input_id="train_a_dataset",
    dataset=train_df_A,
    target_column=demo_dataset.target_column,
)
vm_test_ds_A = vm.init_dataset(
    input_id="test_a_dataset",
    dataset=test_df_A,
    target_column=demo_dataset.target_column,
)

# Initialize training and testing datasets for model B
vm_train_ds_B = vm.init_dataset(
    input_id="train_b_dataset",
    dataset=train_df_B,
    target_column=demo_dataset.target_column,
)
vm_test_ds_B = vm.init_dataset(
    input_id="test_b_dataset",
    dataset=test_df_B,
    target_column=demo_dataset.target_column,
)

# Initialize model A
vm_model_A = vm.init_model(
    input_id="model_a",
    model=model_A,
)

vm_train_ds_A.assign_predictions(model=vm_model_A)
vm_test_ds_A.assign_predictions(model=vm_model_A)

# Initialize model B
vm_model_B = vm.init_model(
    input_id="model_b",
    model=model_B,
)
vm_train_ds_B.assign_predictions(model=vm_model_B)
vm_test_ds_B.assign_predictions(model=vm_model_B)

models = [vm_model_A, vm_model_B]
2024-04-10 17:28:54,767 - INFO(validmind.client): Pandas dataset detected. Initializing VM Dataset instance...
2024-04-10 17:28:54,809 - INFO(validmind.client): Pandas dataset detected. Initializing VM Dataset instance...
2024-04-10 17:28:54,847 - INFO(validmind.client): Pandas dataset detected. Initializing VM Dataset instance...
2024-04-10 17:28:54,915 - INFO(validmind.client): Pandas dataset detected. Initializing VM Dataset instance...
2024-04-10 17:28:55,744 - INFO(validmind.vm_models.dataset): Running predict()... This may take a while
2024-04-10 17:28:55,745 - INFO(validmind.vm_models.dataset): Running predict()... This may take a while
2024-04-10 17:28:55,782 - INFO(validmind.vm_models.dataset): Running predict()... This may take a while
2024-04-10 17:28:55,782 - INFO(validmind.vm_models.dataset): Running predict()... This may take a while

Step 6: Run model validation test suite on models

Explore the time series model validation test suite

vm.test_suites.describe_test_suite("time_series_model_validation")
ID Name Description Tests
time_series_model_validation TimeSeriesModelValidation Test suite for time series model validation. validmind.data_validation.DatasetSplit, validmind.model_validation.ModelMetadata, validmind.model_validation.statsmodels.RegressionModelsCoeffs, validmind.model_validation.statsmodels.RegressionModelsPerformance, validmind.model_validation.statsmodels.RegressionModelForecastPlotLevels, validmind.model_validation.statsmodels.RegressionModelSensitivityPlot

Run model validation test suite on a list of models

config = {
    "validmind.model_validation.statsmodels.RegressionModelForecastPlotLevels": {
        "params": {
            "transformation": "integrate",
        }
    },
    "validmind.model_validation.statsmodels.RegressionModelSensitivityPlot": {
        "params": {
            "transformation": "integrate",
            "shocks": [0.3],
        }
    },
    "validmind.model_validation.statsmodels.RegressionModelsPerformance": {
        "inputs": {
            "in_sample_datasets": (vm_train_ds_A, vm_train_ds_B),
            "out_of_sample_datasets": (vm_test_ds_A, vm_test_ds_B),
            "models": models,
        },
        "params": {
            "transformation": "integrate",
            "shocks": [0.3],
        },
    },
    "validmind.model_validation.statsmodels.RegressionModelForecastPlotLevels": {
        "inputs": {
            "datasets": (vm_train_ds_A, vm_test_ds_A),
            "models": [vm_model_A],
        },
    },
    "validmind.model_validation.statsmodels.RegressionModelSensitivityPlot": {
        "inputs": {
            "datasets": (vm_train_ds_A, vm_test_ds_A),
            "models": [vm_model_A],
        },
    },
}

full_suite = vm.run_test_suite(
    "time_series_model_validation",
    inputs={
        "dataset": vm_train_ds_A,
        "datasets": (vm_train_ds_A, vm_test_ds_A),
        "model": vm_model_A,
        "models": models,
    },
    config=config,
)
2024-04-10 17:28:55,901 - INFO(validmind.tests.model_validation.statsmodels.RegressionModelSensitivityPlot): {'transformation': None, 'shocks': [0.1]}
2024-04-10 17:28:55,901 - INFO(validmind.tests.model_validation.statsmodels.RegressionModelSensitivityPlot): {'transformation': None, 'shocks': [0.1]}

Next steps

You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way: view the prompt validation test results as part of your model documentation right in the ValidMind Platform UI:

  1. In the Platform UI, go to the Documentation page for the model you registered earlier.

  2. Expand 3. Model Development to review all test results.

What you can see now is a more easily consumable version of the prompt validation testing you just performed, along with other parts of your model documentation that still need to be completed.

If you want to learn more about where you are in the model documentation process, take a look at How do I use the framework?.